Machine learning approach to real-time patient motion monitoring

ABSTRACT

Systems and techniques may be used to estimate a patient state during a radiotherapy treatment. For example, a method may include generating a dictionary of expanded potential patient measurements and corresponding potential patient states using a preliminary motion model. The method may include training, using a machine learning technique, a correspondence motion model relating an input patient measurement to an output patient state using the dictionary. The method may include estimating, using a processor, the patient state corresponding to an input image using the correspondence motion model.

TECHNICAL FIELD

Embodiments of the present disclosure pertain generally to medical imageand artificial intelligence processing techniques. In particular, thepresent disclosure pertains to use of machine learning for real-timepatient state estimation.

BACKGROUND

In radiotherapy or radiosurgery, treatment planning is typicallyperformed based on medical images of a patient and requires thedelineation of target volumes and normal critical organs in the medicalimages. One challenge occurs with accurately tracking the variousobjects, such as a tumor, healthy tissue, or other aspects of patientanatomy when the patient is moving (e.g., breathing).

Current techniques are unable to directly measure a changing patientstate in real-time. For example, some techniques use 2D imaging, such as2D kV projections or 2D MRI slices, which are not able to completelytrack the various objects.

Other techniques may rely on detecting surface information, eitherdirectly or by tracking markers on a vest or a box affixed to thepatient. These techniques assume that the surface information iscorrelated to internal patient state, which is often not accurate.

Yet other techniques may rely on implanting markers, such asmagnetically tracked markers, or using x-ray detection of radio-opaquemarkers. These techniques are invasive and correspond only to limitedpoints within the patient.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsdescribe substantially similar components throughout the several views.Like numerals having different letter suffixes represent differentinstances of substantially similar components. The drawings illustrategenerally, by way of example but not by way of limitation, variousembodiments discussed in the present document.

FIG. 1 illustrates an exemplary radiotherapy system adapted forperforming image patient state estimation processing.

FIG. 2 illustrates an exemplary image-guided radiotherapy device.

FIG. 3 illustrates a partially cut-away view of an exemplary systemincluding a combined radiation therapy system and an imaging system,such as a nuclear magnetic resonance (MR) imaging system.

FIG. 4 illustrates an exemplary flow diagram for estimating a patientstate using partial measurements and a preliminary patient model.

FIG. 5 illustrates an exemplary flowchart showing a patient statedictionary generation technique.

FIG. 6 illustrates an exemplary regression model machine learning enginefor use in estimating a patient state.

FIG. 7 illustrates a flowchart of exemplary operations for estimating apatient state.

FIG. 8 illustrates a flowchart of exemplary operations for performingradiation therapy techniques.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings which form a part hereof, and which is shown byway of illustration-specific embodiments in which the present inventionmay be practiced. These embodiments, which are also referred to hereinas “examples,” are described in sufficient detail to enable thoseskilled in the art to practice the invention, and it is to be understoodthat the embodiments may be combined, or that other embodiments may beutilized and that structural, logical and electrical changes may be madewithout departing from the scope of the present invention. The followingdetailed description is, therefore, not be taken in a limiting sense,and the scope of the present invention is defined by the appended claimsand their equivalents.

Image guided radiation therapy (IGRT) is a technique that makes use ofimaging of a patient, in treatment position, immediately prior toirradiation. This allows more accurate targeting of anatomy, such as anorgans, tumors or organs-at-risk. If the patient is expected to moveduring treatment, for example motion caused by breathing which creates aquasi-periodic motion of a lung tumor, or bladder filling causing theprostate position to drift, additional margins may be placed around thetarget to encompass the expected patient motion. These larger marginscome at the expense of high dose to surrounding normal tissue, which maylead to increased side-effects.

IGRT may use computed tomography (CT) imaging, cone beam CT (CBCT),magnetic resonance (MR) imaging, positron-emission tomography (PET)imaging, or the like to obtain a 3D or 4D image of a patient prior toirradiation. For example, a CBCT-enabled linac (linear accelerator) mayconsist of a kV source/detector affixed to the gantry at a 90 degreeangle to a radiation beam, or a MR-Linac device may consist of a linacintegrated directly with an MR scanner.

Localizing motion during the actual irradiation treatment delivery(intrafraction motion) may allow reduction of additional treatmentmargins that would otherwise be used to encompass motion, thus eitherallowing higher doses to be delivered, reduction of side-effects, orboth. Many IGRT imaging technologies are generally not sufficiently fastfor imaging intrafractional motion. For example, CBCT requires multiplekV images from various angles to reconstruct a full 3D patient image,and a 3D MR requires multiple 2D slices, or filling of the full 3Dk-space, each which may take minutes to generate a full 3D image.

In some cases, the real-time or quasi-real-time data that would usuallybe completely acquired prior to generation of a 3D IGRT image, can beused as it is gathered to estimate the instantaneous 3D image at a muchfaster refresh rate from the incomplete, yet fast, stream of incominginformation. For example, 2D kV projections or 2D MR slices may be usedto estimate a full 3D CBCT-like or 3D MR-like image that evolves withthe actual patient motion during treatment. Although fast, on their ownthese 2D images provide only a particular perspective of the patient,not the full 3D picture.

A patient state generator may receive partial measurements (e.g., a 2Dimage) as an input and generate (e.g., estimate) a patient state (e.g.,a 3D image) as an output. To generate a patient state, the generator mayuse a single current partial measurement, a future (predicted) or pastpartial measurement, or a number of partial measurements (e.g., the last10 measurements). These partial measurements may be from a singlemodality, such as an x-ray projection or MRI slice, or from multiplemodalities, such as positions of reflective surface markers on thepatient's surface synchronized with x-ray projections. A patient statemay be a 3D image, or of ‘multi-modality,’ for example the patient statemay include two or more 3D images that offer different information onthe patient state, such as a ‘MR-like’ for enhanced tissue contrast, a‘CT-like’ for high geometric accuracy and voxels related to density thatare useful for dose calculations, or a ‘functional MR-like’ to providefunction information about the patient. Patient state may also includenon-imaging information. A patient state include one or more points ofinterest (such as a target position), contours, surfaces, deformationvector fields, or any information that is relevant to optimizing patienttreatments.

Partial measurements described above may be received in a real-timestream of images (e.g., 2D images) taken from a kV imager or a MRimager, for example. The kV imager may produce stereoscopic 2D imagesfor the real-time stream (e.g., two x-ray images that are orthogonal andacquired substantially simultaneously). The kV imager may be fixed in aroom or coupled to a treatment device (e.g., attached to a gantry). TheMR imager may produce 2D MR slices, which may be orthogonal or parallel.A patient state may be generated from an image or pair of imagesreceived. For example, at any given moment in time, the patient statefor the last received image from the real-time stream may be generated.

In an example, a patient model may be based on data currently collectedin a given fraction, in a pre-treatment phase (after the patient is setup and before the beam is turned on), from another fraction or duringsimulation/planning, using other patients, using generalized patientanatomy, using mechanical models, or any other information that mayassist in defining a patient state from partial measurements. In anexample, the patient model is a 4D dataset, acquired pre-treatment,which represents changes in patient state over a limited period of time(e.g. one representative respiratory cycle). The patient model may betrained, (e.g., using a machine learning technique), to relate an inputpatient measurement (e.g., an image or pair of images from a real-timestream) to an output patient state, for example using a dictionarydefining constructed patient measurements to corresponding patientstates. The patient model may be warped by a deformation vector field(DVF) as a function of one or more parameters to generate a patientstate.

The patient model in a 4D dataset may include the patient state thatvaries with a single parameter, such as phase in a respiratory cycle.The patient model may be used to build a time-varying patient state overa representative breathing cycle, which may treat each breath as more orless the same. This simplifies the modeling by allowing chunks ofpartial imaging data to be taken from different breathing cycles andassigned to a single representative breathing cycle. A 3D image may thenbe reconstructed for each phase ‘bin’.

In an example, the patient state may be represented, for example, as a3D image, or a 3D DVF plus a 3D reference image. These may beequivalent, since the elements of the 3D DVF and the 3D reference imagemay be used to obtain (e.g., deform the 3D reference image with the 3DDVF) the 3D image.

FIG. 1 illustrates an exemplary radiotherapy system adapted forperforming patient state estimation processing. This patient stateestimation processing is performed to enable the radiotherapy system toprovide radiation therapy to a patient based on specific aspects ofcaptured medical imaging data. The radiotherapy system includes an imageprocessing computing system 110 which hosts patient state processinglogic 120. The image processing computing system 110 may be connected toa network (not shown), and such network may be connected to theInternet. For instance, a network can connect the image processingcomputing system 110 with one or more medical information sources (e.g.,a radiology information system (RIS), a medical record system (e.g., anelectronic medical record (EMR)/electronic health record (EHR) system),an oncology information system (OIS)), one or more image data sources150, an image acquisition device 170, and a treatment device 180 (e.g.,a radiation therapy device). As an example, the image processingcomputing system 110 can be configured to perform image patient stateoperations by executing instructions or data from the patient stateprocessing logic 120, as part of operations to generate and customizeradiation therapy treatment plans to be used by the treatment device180.

The image processing computing system 110 may include processingcircuitry 112, memory 114, a storage device 116, and other hardware andsoftware-operable features such as a user interface 140, communicationinterface, and the like. The storage device 116 may storecomputer-executable instructions, such as an operating system, radiationtherapy treatment plans (e.g., original treatment plans, adaptedtreatment plans, or the like), software programs (e.g., radiotherapytreatment plan software, artificial intelligence implementations such asdeep learning models, machine learning models, and neural networks,etc.), and any other computer-executable instructions to be executed bythe processing circuitry 112.

In an example, the processing circuitry 112 may include a processingdevice, such as one or more general-purpose processing devices such as amicroprocessor, a central processing unit (CPU), a graphics processingunit (GPU), an accelerated processing unit (APU), or the like. Moreparticularly, the processing circuitry 112 may be a complex instructionset computing (CISC) microprocessor, a reduced instruction set computing(RISC) microprocessor, a very long instruction Word (VLIW)microprocessor, a processor implementing other instruction sets, orprocessors implementing a combination of instruction sets. Theprocessing circuitry 112 may also be implemented by one or morespecial-purpose processing devices such as an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA), adigital signal processor (DSP), a System on a Chip (SoC), or the like.As would be appreciated by those skilled in the art, in some examples,the processing circuitry 112 may be a special-purpose processor, ratherthan a general-purpose processor. The processing circuitry 112 mayinclude one or more known processing devices, such as a microprocessorfrom the Pentium™, Core™, Xeon™, or Itanium® family manufactured byIntel™, the Turion™, Athlon™, Sempron™, Opteron™, FX™, Phenom™ familymanufactured by AMD™, or any of various processors manufactured by SunMicrosystems. The processing circuitry 112 may also include graphicalprocessing units such as a GPU from the GeForce®, Quadro®, Tesla® familymanufactured by Nvidia™, GMA, Iris™ family manufactured by Intel™, orthe Radeon™ family manufactured by AMD™. The processing circuitry 112may also include accelerated processing units such as the Xeon Phi™family manufactured by Intel™. The disclosed embodiments are not limitedto any type of processor(s) otherwise configured to meet the computingdemands of identifying, analyzing, maintaining, generating, and/orproviding large amounts of data or manipulating such data to perform themethods disclosed herein. In addition, the term “processor” may includemore than one processor, for example, a multi-core design or a pluralityof processors each having a multi-core design. The processing circuitry112 can execute sequences of computer program instructions, stored inmemory 114, and accessed from the storage device 116, to perform variousoperations, processes, methods that will be explained in greater detailbelow.

The memory 114 may comprise read-only memory (ROM), a phase-changerandom access memory (PRAM), a static random access memory (SRAM), aflash memory, a random access memory (RAM), a dynamic random accessmemory (DRAM) such as synchronous DRAM (SDRAM), an electrically erasableprogrammable read-only memory (EEPROM), a static memory (e.g., flashmemory, flash disk, static random access memory) as well as other typesof random access memories, a cache, a register, a compact disc read-onlymemory (CD-ROM), a digital versatile disc (DVD) or other opticalstorage, a cassette tape, other magnetic storage device, or any othernon-transitory medium that may be used to store information includingimage, data, or computer executable instructions (e.g., stored in anyformat) capable of being accessed by the processing circuitry 112, orany other type of computer device. For instance, the computer programinstructions can be accessed by the processing circuitry 112, read fromthe ROM, or any other suitable memory location, and loaded into the RAMfor execution by the processing circuitry 112.

The storage device 116 may constitute a drive unit that includes amachine-readable medium on which is stored one or more sets ofinstructions and data structures (e.g., software) embodying or utilizedby any one or more of the methodologies or functions described herein(including, in various examples, the patient state processing logic 120and the user interface 140). The instructions may also reside,completely or at least partially, within the memory 114 and/or withinthe processing circuitry 112 during execution thereof by the imageprocessing computing system 110, with the memory 114 and the processingcircuitry 112 also constituting machine-readable media.

The memory device 114 or the storage device 116 may constitute anon-transitory computer-readable medium. For example, the memory device114 or the storage device 116 may store or load instructions for one ormore software applications on the computer-readable medium. Softwareapplications stored or loaded with the memory device 114 or the storagedevice 116 may include, for example, an operating system for commoncomputer systems as well as for software-controlled devices. The imageprocessing computing system 110 may also operate a variety of softwareprograms comprising software code for implementing the patient stateprocessing logic 120 and the user interface 140. Further, the memorydevice 114 and the storage device 116 may store or load an entiresoftware application, part of a software application, or code or datathat is associated with a software application, which is executable bythe processing circuitry 112. In a further example, the memory device114 or the storage device 116 may store, load, or manipulate one or moreradiation therapy treatment plans, imaging data, patient state data,dictionary entries, artificial intelligence model data, labels andmapping data, etc. It is contemplated that software programs may bestored not only on the storage device 116 and the memory 114 but also ona removable computer medium, such as a hard drive, a computer disk, aCD-ROM, a DVD, a HD, a Blu-Ray DVD, USB flash drive, a SD card, a memorystick, or any other suitable medium; such software programs may also becommunicated or received over a network.

Although not depicted, the image processing computing system 110 mayinclude a communication interface, network interface card, andcommunications circuitry. An example communication interface mayinclude, for example, a network adaptor, a cable connector, a serialconnector, a USB connector, a parallel connector, a high-speed datatransmission adaptor (e.g., such as fiber, USB 3.0, thunderbolt, and thelike), a wireless network adaptor (e.g., such as a IEEE 802.11/Wi-Fiadapter), a telecommunication adapter (e.g., to communicate with 3G,4G/LTE, and 5G, networks and the like), and the like. Such acommunication interface may include one or more digital and/or analogcommunication devices that permit a machine to communicate with othermachines and devices, such as remotely located components, via anetwork. The network may provide the functionality of a local areanetwork (LAN), a wireless network, a cloud computing environment (e.g.,software as a service, platform as a service, infrastructure as aservice, etc.), a client-server, a wide area network (WAN), and thelike. For example, network may be a LAN or a WAN that may include othersystems (including additional image processing computing systems orimage-based components associated with medical imaging or radiotherapyoperations).

In an example, the image processing computing system 110 may obtainimage data 160 from the image data source 150, for hosting on thestorage device 116 and the memory 114. In an example, the softwareprograms operating on the image processing computing system 110 mayconvert medical images of one format (e.g., MRI) to another format(e.g., CT), such as by producing synthetic images, such as a pseudo-CTimage. In another example, the software programs may register orassociate a patient medical image (e.g., a CT image or an MR image) withthat patient's dose distribution of radiotherapy treatment (e.g., alsorepresented as an image) so that corresponding image voxels and dosevoxels are appropriately associated. In yet another example, thesoftware programs may substitute functions of the patient images such assigned distance functions or processed versions of the images thatemphasize some aspect of the image information. Such functions mightemphasize edges or differences in voxel textures, or other structuralaspects. In another example, the software programs may visualize, hide,emphasize, or de-emphasize some aspect of anatomical features, patientmeasurements, patient state information, or dose or treatmentinformation, within medical images. The storage device 116 and memory114 may store and host data to perform these purposes, including theimage data 160, patient data, and other data required to create andimplement a radiation therapy treatment plan and associated patientstate estimation operations.

The processing circuitry 112 may be communicatively coupled to thememory 114 and the storage device 116, and the processing circuitry 112may be configured to execute computer executable instructions storedthereon from either the memory 114 or the storage device 116. Theprocessing circuitry 112 may execute instructions to cause medicalimages from the image data 160 to be received or obtained in memory 114,and processed using the patient state processing logic 120. For example,the image processing computing system 110 may receive image data 160from the image acquisition device 170 or image data sources 150 via acommunication interface and network to be stored or cached in thestorage device 116. The processing circuitry 112 may also send or updatemedical images stored in memory 114 or the storage device 116 via acommunication interface to another database or data store (e.g., amedical facility database). In some examples, one or more of the systemsmay form a distributed computing/simulation environment that uses anetwork to collaboratively perform the embodiments described herein. Inaddition, such network may be connected to internet to communicate withservers and clients that reside remotely on the internet.

In further examples, the processing circuitry 112 may utilize softwareprograms (e.g., a treatment planning software) along with the image data160 and other patient data to create a radiation therapy treatment plan.In an example, the image data 160 may include 2D or 3D images, such asfrom a CT or MR. In addition, the processing circuitry 112 may utilizesoftware programs to generate an estimated patient state from adictionary of measurements and corresponding patient states, such asusing a correspondence motion model and a machine learning algorithm(e.g., a regression algorithm).

Further, such software programs may utilize patient state processinglogic 120 to implement a patient state estimation workflow 130, usingthe techniques further discussed herein. The processing circuitry 112may subsequently then transmit the executable radiation therapytreatment plan via a communication interface and the network to thetreatment device 180, where the radiation therapy plan will be used totreat a patient with radiation via the treatment device, consistent withresults of the patient state estimation workflow 130. Other outputs anduses of the software programs and the patient state estimation workflow130 may occur with use of the image processing computing system 110.

As discussed herein (e.g., with reference to the patient stateestimation discussed herein, the processing circuitry 112 may executesoftware programs that invokes the patient state processing logic 120 toimplement functions including generation of a preliminary motion model,creation of a dictionary, training a patient state generator usingmachine learning, patient state estimation, and other aspects ofautomatic processing and artificial intelligence. For instance, theprocessing circuitry 112 may execute software programs that estimate apatient state using a machine learning trained system.

In an example, the image data 160 may include one or more MRI images(e.g., 2D MRI, 3D MRI, 2D streaming MRI, 4D MRI, 4D volumetric MRI, 4Dcine MRI, etc.), functional MRI images (e.g., fMRI, DCE-MRI, diffusionMM), Computed Tomography (CT) images (e.g., 2D CT, Cone beam CT, 3D CT,4D CT), ultrasound images (e.g., 2D ultrasound, 3D ultrasound, 4Dultrasound), Positron Emission Tomography (PET) images, X-ray images,fluoroscopic images, radiotherapy portal images, Single-Photo EmissionComputed Tomography (SPECT) images, computer generated synthetic images(e.g., pseudo-CT images) and the like. Further, the image data 160 mayalso include or be associated with medical image processing data, forinstance, training images, and ground truth images, contoured images,and dose images. In an example, the image data 160 may be received fromthe image acquisition device 170 and stored in one or more of the imagedata sources 150 (e.g., a Picture Archiving and Communication System(PACS), a Vendor Neutral Archive (VNA), a medical record or informationsystem, a data warehouse, etc.). Accordingly, the image acquisitiondevice 170 may comprise a MRI imaging device, a CT imaging device, a PETimaging device, an ultrasound imaging device, a fluoroscopic device, aSPECT imaging device, an integrated Linear Accelerator and MRI imagingdevice, or other medical imaging devices for obtaining the medicalimages of the patient. The image data 160 may be received and stored inany type of data or any type of format (e.g., in a Digital Imaging andCommunications in Medicine (DICOM) format) that the image acquisitiondevice 170 and the image processing computing system 110 may use toperform operations consistent with the disclosed embodiments.

In an example, the image acquisition device 170 may be integrated withthe treatment device 180 as a single apparatus (e.g., a MRI devicecombined with a linear accelerator, also referred to as an “MR-linac”,as shown and described in FIG. 3 below). Such an MR-linac can be used,for example, to precisely determine a location of a target organ or atarget tumor in the patient, so as to direct radiation therapyaccurately according to the radiation therapy treatment plan to apredetermined target. For instance, a radiation therapy treatment planmay provide information about a particular radiation dose to be appliedto each patient. The radiation therapy treatment plan may also includeother radiotherapy information, such as beam angles,dose-histogram-volume information, the number of radiation beams to beused during therapy, the dose per beam, and the like.

The image processing computing system 110 may communicate with anexternal database through a network to send/receive a plurality ofvarious types of data related to image processing and radiotherapyoperations. For example, an external database may include machine datathat is information associated with the treatment device 180, the imageacquisition device 170, or other machines relevant to radiotherapy ormedical procedures. Machine data information may include radiation beamsize, arc placement, beam on and off time duration, machine parameters,segments, multi-leaf collimator (MLC) configuration, gantry speed, MRIpulse sequence, and the like. The external database may be a storagedevice and may be equipped with appropriate database administrationsoftware programs. Further, such databases or data sources may include aplurality of devices or systems located either in a central or adistributed manner.

The image processing computing system 110 can collect and obtain data,and communicate with other systems, via a network using one or morecommunication interfaces, which are communicatively coupled to theprocessing circuitry 112 and the memory 114. For instance, acommunication interface may provide communication connections betweenthe image processing computing system 110 and radiotherapy systemcomponents (e.g., permitting the exchange of data with externaldevices). For instance, the communication interface may in some exampleshave appropriate interfacing circuitry from an output device 142 or aninput device 144 to connect to the user interface 140, which may be ahardware keyboard, a keypad, or a touch screen through which a user mayinput information into the radiotherapy system.

As an example, the output device 142 may include a display device whichoutputs a representation of the user interface 140 and one or moreaspects, visualizations, or representations of the medical images. Theoutput device 142 may include one or more display screens that displaymedical images, interface information, treatment planning parameters(e.g., contours, dosages, beam angles, labels, maps, etc.) treatmentplans, a target, localizing a target or tracking a target, patient stateestimations (e.g., a 3D image), or any related information to the user.The input device 144 connected to the user interface 140 may be akeyboard, a keypad, a touch screen or any type of device that a user mayinput information to the radiotherapy system. Alternatively, the outputdevice 142, the input device 144, and features of the user interface 140may be integrated into a single device such as a smartphone or tabletcomputer, e.g., Apple iPad®, Lenovo Thinkpad®, Samsung Galaxy®, etc.

Furthermore, any and all components of the radiotherapy system may beimplemented as a virtual machine (e.g., via VMWare, Hyper-V, and thelike virtualization platforms). For instance, a virtual machine can besoftware that functions as hardware. Therefore, a virtual machine caninclude at least one or more virtual processors, one or more virtualmemories, and one or more virtual communication interfaces that togetherfunction as hardware. For example, the image processing computing system110, the image data sources 150, or like components, may be implementedas a virtual machine or within a cloud-based virtualization environment.

The patient state processing logic 120 or other software programs maycause the computing system to communicate with the image data sources150 to read images into memory 114 and the storage device 116, or storeimages or associated data from the memory 114 or the storage device 116to and from the image data sources 150. For example, the image datasource 150 may be configured to store and provide a plurality of images(e.g., 3D MRI, 4D MRI, 2D MRI slice images, CT images, 2D Fluoroscopyimages, X-ray images, raw data from MR scans or CT scans, DigitalImaging and Communications in Medicine (DICOM) metadata, etc.) that theimage data source 150 hosts, from image sets in image data 160 obtainedfrom one or more patients via the image acquisition device 170. Theimage data source 150 or other databases may also store data to be usedby the patient state processing logic 120 when executing a softwareprogram that performs patient state estimation operations, or whencreating radiation therapy treatment plans. Further, various databasesmay store the data produced by the preliminary motion model (such as thedictionary), the correspondence motion model, or machine learningmodels, including the network parameters constituting the model learnedby the network and the resulting predicted data. The image processingcomputing system 110 thus may obtain and/or receive the image data 160(e.g., 2D MRI slice images, CT images, 2D Fluoroscopy images, X-rayimages, 3D MRI images, 4D MRI images, etc.) from the image data source150, the image acquisition device 170, the treatment device 180 (e.g., aMRI-Linac), or other information systems, in connection with performingimage patient state estimation as part of treatment or diagnosticoperations.

The image acquisition device 170 can be configured to acquire one ormore images of the patient's anatomy for a region of interest (e.g., atarget organ, a target tumor or both). Each image, typically a 2D imageor slice, can include one or more parameters (e.g., a 2D slicethickness, an orientation, and a location, etc.). In an example, theimage acquisition device 170 can acquire a 2D slice in any orientation.For example, an orientation of the 2D slice can include a sagittalorientation, a coronal orientation, or an axial orientation. Theprocessing circuitry 112 can adjust one or more parameters, such as thethickness and/or orientation of the 2D slice, to include the targetorgan and/or target tumor. In an example, 2D slices can be determinedfrom information such as a 3D MRI volume. Such 2D slices can be acquiredby the image acquisition device 170 in “real-time” while a patient isundergoing radiation therapy treatment, for example, when using thetreatment device 180 (with “real-time” meaning acquiring the data in 10milliseconds or less). In another example for some applications,real-time may include a timeframe within (e.g., up to) 200 or 300milliseconds. In an example, real-time may include a time period fastenough for a clinical problem being solved by techniques describedherein. In this example, real-time may vary depending on target speed,radiotherapy margins, lag, response time of a treatment device, etc.

The patient state processing logic 120 in the image processing computingsystem 110 is depicted as implementing a patient state estimationworkflow 130 with various aspects of model generation and estimationprocessing operations. In an example, the patient state estimationworkflow 130 operated by the patient state processing logic 120generates and uses a preliminary motion model 132 generated from patientdata (e.g., from a patient being treated, from multiple previouspatients, or the like). The preliminary motion model 132 may include amodel of a patient under motion (e.g., breathing) generated based onpatient measurements and corresponding patient states. The patient stateestimation workflow 130 includes creation of a dictionary 134 by usingthe preliminary motion model to generate sample (potential) patientmeasurements and corresponding patient states. The patient stateestimation workflow 130 includes training a correspondence motion modelusing machine learning 136 (e.g., using a regression-based machinelearning technique) based on the dictionary 134. The patient stateestimation workflow 130 includes estimating a patient state 138 usingthe correspondence motion model and a current patient measurement (e.g.,2D image).

The patient state processing logic 120 and the patient state estimationworkflow 130 may be used when generating the radiation therapy treatmentplan, within use of software programs such as treatment planningsoftware, such as Monaco®, manufactured by Elekta AB of Stockholm,Sweden. In order to generate the radiation therapy treatment plans, theimage processing computing system 110 may communicate with the imageacquisition device 170 (e.g., a CT device, a MRI device, a PET device,an X-ray device, an ultrasound device, etc.) to capture and accessimages of the patient and to delineate a target, such as a tumor. Insome examples, the delineation of one or more organs at risk (OARs),such as healthy tissue surrounding the tumor or in close proximity tothe tumor may be required.

In order to delineate a target organ or a target tumor from the OAR,medical images, such as MRI images, CT images, PET images, fMRI images,X-ray images, ultrasound images, radiotherapy portal images, SPECTimages and the like, of the patient undergoing radiotherapy may beobtained non-invasively by the image acquisition device 170 to revealthe internal structure of a body part. Based on the information from themedical images, a 3D structure of the relevant anatomical portion may beobtained. In addition, during a treatment planning process, manyparameters may be taken into consideration to achieve a balance betweenefficient treatment of the target tumor (e.g., such that the targettumor receives enough radiation dose for an effective therapy) and lowirradiation of the OAR(s) (e.g., the OAR(s) receives as low a radiationdose as possible), for example by using an estimated patient state todetermine where OAR(s) may be at a given time when the patient is moving(e.g., breathing). Other parameters that may be considered include thelocation of the target organ and the target tumor, the location of theOAR, and the movement of the target in relation to the OAR. For example,the 3D structure may be obtained by contouring the target or contouringthe OAR within each 2D layer or slice of an MRI or CT image andcombining the contour of each 2D layer or slice. The contour may begenerated manually (e.g., by a physician, dosimetrist, or health careworker using a program such as Monaco® manufactured by Elekta AB ofStockholm, Sweden) or automatically (e.g., using a program such as theAtlas-based auto-segmentation software, ABAS®, manufactured by Elekta ABof Stockholm, Sweden).

After the target tumor and the OAR(s) have been located and delineated,a dosimetrist, physician or healthcare worker may determine a dose ofradiation to be applied to the target tumor, as well as any maximumamounts of dose that may be received by the OAR proximate to the tumor(e.g., left and right parotid, optic nerves, eyes, lens, inner ears,spinal cord, brain stem, and the like). After the radiation dose isdetermined for each anatomical structure (e.g., target tumor, OAR), aprocess known as inverse planning may be performed to determine one ormore treatment plan parameters that would achieve the desired radiationdose distribution. Examples of treatment plan parameters include volumedelineation parameters (e.g., which define target volumes, contoursensitive structures, etc.), margins around the target tumor and OARs,beam angle selection, collimator settings, and beam-on times. During theinverse-planning process, the physician may define dose constraintparameters that set bounds on how much radiation an OAR may receive(e.g., defining full dose to the tumor target and zero dose to any OAR;defining 95% of dose to the target tumor; defining that the spinal cord,brain stem, and optic structures receive ≤45Gy, ≤55Gy and <54Gy,respectively). The result of inverse planning may constitute a radiationtherapy treatment plan that may be stored. Some of these treatmentparameters may be correlated. For example, tuning one parameter (e.g.,weights for different objectives, such as increasing the dose to thetarget tumor) in an attempt to change the treatment plan may affect atleast one other parameter, which in turn may result in the developmentof a different treatment plan. Thus, the image processing computingsystem 110 can generate a tailored radiation therapy treatment planhaving these parameters in order for the treatment device 180 to providesuitable radiotherapy treatment to the patient.

FIG. 2 illustrates an exemplary image-guided radiotherapy device 202,that includes include a radiation source, such as an X-ray source or alinear accelerator, a couch 216, an imaging detector 214, and aradiation therapy output 204. The radiation therapy device 202 may beconfigured to emit a radiation beam 208 to provide therapy to a patient.The radiation therapy output 204 can include one or more attenuators orcollimators, such as a multi-leaf collimator (MLC).

As an example, a patient can be positioned in a region 212, supported bythe treatment couch 216 to receive a radiation therapy dose according toa radiation therapy treatment plan (e.g., a treatment plan generated bythe radiotherapy system of FIG. 1). The radiation therapy output 204 canbe mounted or attached to a gantry 206 or other mechanical support. Oneor more chassis motors (not shown) may rotate the gantry 206 and theradiation therapy output 204 around couch 216 when the couch 216 isinserted into the treatment area. In an example, gantry 206 may becontinuously rotatable around couch 216 when the couch 216 is insertedinto the treatment area. In another example, gantry 206 may rotate to apredetermined position when the couch 216 is inserted into the treatmentarea. For example, the gantry 206 can be configured to rotate thetherapy output 204 around an axis (“A”). Both the couch 216 and theradiation therapy output 204 can be independently moveable to otherpositions around the patient, such as moveable in transverse direction(“T”), moveable in a lateral direction (“L”), or as rotation about oneor more other axes, such as rotation about a transverse axis (indicatedas “R”). A controller communicatively connected to one or more actuators(not shown) may control the couch 216 movements or rotations in order toproperly position the patient in or out of the radiation beam 208according to a radiation therapy treatment plan. As both the couch 216and the gantry 206 are independently moveable from one another inmultiple degrees of freedom, which allows the patient to be positionedsuch that the radiation beam 208 precisely can target the tumor.

The coordinate system (including axes A, T, and L) shown in FIG. 2 canhave an origin located at an isocenter 210. The isocenter can be definedas a location where the central axis of the radiation therapy beam 208intersects the origin of a coordinate axis, such as to deliver aprescribed radiation dose to a location on or within a patient.Alternatively, the isocenter 210 can be defined as a location where thecentral axis of the radiation therapy beam 208 intersects the patientfor various rotational positions of the radiation therapy output 204 aspositioned by the gantry 206 around the axis A.

Gantry 206 may also have an attached imaging detector 214. The imagingdetector 214 is preferably located opposite to the radiation source(output 204), and in an example, the imaging detector 214 can be locatedwithin a field of the therapy beam 208.

The imaging detector 214 can be mounted on the gantry 206 preferablyopposite the radiation therapy output 204, such as to maintain alignmentwith the therapy beam 208. The imaging detector 214 rotating about therotational axis as the gantry 206 rotates. In an example, the imagingdetector 214 can be a flat panel detector (e.g., a direct detector or ascintillator detector). In this manner, the imaging detector 214 can beused to monitor the therapy beam 208 or the imaging detector 214 can beused for imaging the patient's anatomy, such as portal imaging. Thecontrol circuitry of radiation therapy device 202 may be integratedwithin the radiotherapy system or remote from it.

In an illustrative example, one or more of the couch 216, the therapyoutput 204, or the gantry 206 can be automatically positioned, and thetherapy output 204 can establish the therapy beam 208 according to aspecified dose for a particular therapy delivery instance. A sequence oftherapy deliveries can be specified according to a radiation therapytreatment plan, such as using one or more different orientations orlocations of the gantry 206, couch 216, or therapy output 204. Thetherapy deliveries can occur sequentially, but can intersect in adesired therapy locus on or within the patient, such as at the isocenter210. A prescribed cumulative dose of radiation therapy can thereby bedelivered to the therapy locus while damage to tissue nearby the therapylocus can be reduced or avoided.

Thus, FIG. 2 specifically illustrates an example of a radiation therapydevice 202 operable to provide radiotherapy treatment to a patient, witha configuration where a radiation therapy output can be rotated around acentral axis (e.g., an axis “A”). Other radiation therapy outputconfigurations can be used. For example, a radiation therapy output canbe mounted to a robotic arm or manipulator having multiple degrees offreedom. In yet another example, the therapy output can be fixed, suchas located in a region laterally separated from the patient, and aplatform supporting the patient can be used to align a radiation therapyisocenter with a specified target locus within the patient. In anotherexample, a radiation therapy device can be a combination of a linearaccelerator and an image acquisition device. In some examples, the imageacquisition device may be an MRI, an X-ray, a CT, a CBCT, a spiral CT, aPET, a SPECT, an optical tomography, a fluorescence imaging, ultrasoundimaging, an MR-linac, or radiotherapy portal imaging device, etc., aswould be recognized by one of ordinary skill in the art.

FIG. 3 depicts an exemplary radiation therapy system 300 (e.g., known inthe art as a MR-Linac) that can include combining a radiation therapydevice 202 and an imaging system, such as a nuclear magnetic resonance(MR) imaging system consistent with the disclosed embodiments. As shown,system 300 may include a couch 310, an image acquisition device 320, anda radiation delivery device 330. System 300 delivers radiation therapyto a patient in accordance with a radiotherapy treatment plan. In someembodiments, image acquisition device 320 may correspond to imageacquisition device 170 in FIG. 1 that may acquire images.

Couch 310 may support a patient (not shown) during a treatment session.In some implementations, couch 310 may move along a horizontal,translation axis (labelled “I”), such that couch 310 may move thepatient resting on couch 310 into or out of system 300. Couch 310 mayalso rotate around a central vertical axis of rotation, transverse tothe translation axis. To allow such movement or rotation, couch 310 mayhave motors (not shown) enabling the couch to move in various directionsand to rotate along various axes. A controller (not shown) may controlthese movements or rotations in order to properly position the patientaccording to a treatment plan.

In some embodiments, image acquisition device 320 may include an MRImachine used to acquire 2D or 3D MRI images of the patient before,during, or after a treatment session. Image acquisition device 320 mayinclude a magnet 321 for generating a primary magnetic field formagnetic resonance imaging. The magnetic field lines generated byoperation of magnet 321 may run substantially parallel to the centraltranslation axis I. Magnet 321 may include one or more coils with anaxis that runs parallel to the translation axis I. In some embodiments,the one or more coils in magnet 321 may be spaced such that a centralwindow 323 of magnet 321 is free of coils. In other embodiments, thecoils in magnet 321 may be thin enough or of a reduced density such thatthey are substantially transparent to radiation of the wavelengthgenerated by radiotherapy device 330. Image acquisition device 320 mayalso include one or more shielding coils, which may generate a magneticfield outside magnet 321 of approximately equal magnitude and oppositepolarity in order to cancel or reduce any magnetic field outside ofmagnet 321. As described below, radiation source 331 of radiotherapydevice 330 may be positioned in the region where the magnetic field iscancelled, at least to a first order, or reduced.

Image acquisition device 320 may also include two gradient coils 325 and326, which may generate a gradient magnetic field that is superposed onthe primary magnetic field. Coils 325 and 326 may generate a gradient inthe resultant magnetic field that allows spatial encoding of the protonsso that their position can be determined. Gradient coils 325 and 326 maybe positioned around a common central axis with the magnet 321, and maybe displaced along that central axis. The displacement may create a gap,or window, between coils 325 and 326. In the embodiments where magnet321 also includes a central window 323 between coils, the two windowsmay be aligned with each other.

Image acquisition is used to track tumor movement. At times, internal orexternal surrogates may be used. However, implanted seeds may move fromtheir initial positions or become dislodged during radiation therapytreatment. Also, using surrogates assumes there is a correlation betweentumor motion and the displacement of the external surrogate. However,there may be phase shifts between external surrogates and tumor motion,and their positions may frequently lose correlation over time. It isknown that there may be mismatches between tumor and surrogates upwardof 9 mm. Further, any deformation of the shape of a tumor is unknownduring tracking.

An advantage of magnetic resonance imaging (MRI) is in the superior softtissue contrast that is provided to visualize the tumor in more detail.Using a plurality of intrafractional MR images allows the determinationof both shape and position (e.g., centroid) of a tumor. In addition, MMimages improve any manual contouring performed by, for example, aradiation oncologist, even when auto-contouring software (e.g., ABAS®)is utilized. This is because of the high contrast between the tumortarget and the background region provided by MR images.

Another advantage of using an MR-Linac system is that a treatment beamcan be continuously on and thereby executing intrafractional tracking ofthe target tumor. For instance, optical tracking devices or stereoscopicx-ray fluoroscopy systems can detect tumor position at 30 Hz by usingtumor surrogates. With MRI, the imaging acquisition rates are faster(e.g., 3-6 fps). Therefore, the centroid position of the target may bedetermined, artificial intelligence (e.g., neural network) software canpredict a future target position. An added advantage of intrafractionaltracking by using an MR-Linac is that the by being able to predict afuture target location, the leaves of the multi-leaf collimator (MLC)will be able to conform to the target contour a its predicted futureposition. Thus, predicting future tumor position using MRI occurs at thesame rate as imaging frequency during tracking. By being able to trackthe movement of a target tumor clearly using detailed MRI imaging allowsfor the delivery of a highly conformal radiation dose to the movingtarget.

In some embodiments, image acquisition device 320 may be an imagingdevice other than an MRI, such as an X-ray, a CT, a CBCT, a spiral CT, aPET, a SPECT, an optical tomography, a fluorescence imaging, ultrasoundimaging, or radiotherapy portal imaging device, etc. As would berecognized by one of ordinary skill in the art, the above description ofimage acquisition device 320 concerns certain embodiments and is notintended to be limiting.

Radiotherapy device 330 may include the source of radiation 331, such asan X-ray source or a linear accelerator, and a multi-leaf collimator(MLC) 333. Radiotherapy device 330 may be mounted on a chassis 335. Oneor more chassis motors (not shown) may rotate chassis 335 around couch310 when couch 310 is inserted into the treatment area. In anembodiment, chassis 335 may be continuously rotatable around couch 310,when couch 310 is inserted into the treatment area. Chassis 335 may alsohave an attached radiation detector (not shown), preferably locatedopposite to radiation source 331 and with the rotational axis of chassis335 positioned between radiation source 331 and the detector. Further,device 330 may include control circuitry (not shown) used to control,for example, one or more of couch 310, image acquisition device 320, andradiotherapy device 330. The control circuitry of radiotherapy device330 may be integrated within system 300 or remote from it.

During a radiotherapy treatment session, a patient may be positioned oncouch 310. System 300 may then move couch 310 into the treatment areadefined by magnetic coils 321, 325, 326, and chassis 335. Controlcircuitry may then control radiation source 331, MLC 333, and thechassis motor(s) to deliver radiation to the patient through the windowbetween coils 325 and 326 according to a radiotherapy treatment plan.

FIG. 4 illustrates an exemplary flow diagram for estimating a patientstate. FIG. 4 includes a patient state generator 408 for estimating apatient state using a correspondence motion model. The patient stategenerator 408 uses an instantaneous partial measurement 402 and apreliminary motion model of a patient 406 to estimate a patient state,output at block 410. The preliminary motion model 406 is generated usingprevious measurements 404, including previous patient statescorresponding to the previous measurements 404.

In practical radiotherapy applications, partial measurements (e.g., a 2Dimage or image slice) provide incomplete information about the patientstate (e.g., a 3D image). For example, a 2D MRI slice is a single cutthrough a 3D representation of the patient, and an x-ray projection isan integration through voxels along ray-lines of a 3D representation.Using either image results in impartial information (e.g., a 2D imagerather than a 3D representation of patient anatomy). The patient stategenerator 408 may use the partial information and the patient model 406generated from past measurements and/or offline (pre-treatment)acquisitions to estimate the patient state 410.

The patient model generator 408 may include creation of a lowdimensional patient state representation. In an example, priormeasurements are first reconstructed into a 4D image. Examples mayinclude a 4D CT acquired during a planning phase, which is used togenerate a treatment plan; a 4D CBCT acquired immediately prior to eachtreatment session, with the patient in treatment position, generated forexample by rotating a kV imager around the patient on a conventionallinac; a 4D MR acquired prior to each treatment session on an MR-linac,or the like.

4D images may include a series of 3D images of a representativerespiratory cycle. For example, for a 4D CBCT, a number of x-rayprojections are acquired and sorted into a number of bins. Sorting maybe done, for example, by detecting a diaphragm position in eachprojection directly in the images, or using a separate respiratorysignal acquired simultaneously with the kV projections, and binning theprojection according to the phase or amplitude of the signal. Each binis then reconstructed separately with the kV projections assigned tothat bin to form a 3D image per bin. Similar techniques may be used togenerate a 4D MR image. A model may then be constructed using the 4Dimage as an interim step.

In an example, a reference phase of the 4D image is selected (forexample, the one used for treatment planning), and a deformable imageregistration (DIR) is performed between the 3D image of each phase andthat of the reference phase. The reference phase may include high-leveltreatment information (e.g. GTV, organs at risk, etc.). The output ofthe DIR process may include a displacement vector field (DVF) linkingeach phase to the reference phase.

Such a DVF-based motion model provides a mechanism for deforming thereference patient state (e.g., treatment information as defined on the3D reference image) to the specific anatomy exhibited in each of theother phases of the representative respiratory cycle represented in the4D dataset.

To interpolate and even extrapolate the preliminary motion model 406 togenerate new DVFs, an unsupervised dimensionality reduction techniquesuch as principal component analysis (PCA), independent componentanalysis (ICA), canonical correlation analysis (CCA), or the like may beused to identify one or more major degrees of freedom of the respiratorymotion. In an example, 2 or 3 degrees of freedom may be sufficient toaccurately estimate the patient state. In this example, additionaldegrees of freedom may be ignored or discarded (e.g., when they providelittle useful information and are mostly noise). For example, a PCA ofthe DVF motion model may yield a low-dimensionality patient motion modelthat corresponds to a mean DVF and 2 or 3 DVF eigenmodes' (e.g.,weighted inputs representing a degree of freedom). A DVF at any point intime in the motion cycle can be expressed as a weighted sum of the meanand the eigenmodes. For example, the mean DVF may be represented by DVF₀and the eigenmodes may be DVF₁ and DVF₂, which are two full 3D vectorfields, and then the DVF at any time during the cycle can be written asDVF=DVF₀+a₁*DVF₁+a₂*DVF₂ where a₁ and a₂ are scalar numbers andrepresent time variation. In this example, the motion model is reducedto identifying a₁ and a₂ instead of the entire DVF at a particular time.Once calculated, the DVF can then be used to warp the reference 3D imageto obtain the current 3D image representing the patient state 310 (andcan be extended to multiple patient images).

In some cases the interim step of reconstructing a 4D image may not benecessary, and the low dimensional state representation may be createddirectly from the measurements.

In an example, an advantage of using a pre-treatment 4D image is thatthe data is likely to be an excellent representation of the patient'srespiratory degrees of freedom since it was acquired immediately priorto treatment. In some cases there may be advantages to using 4D imagesfrom a previous day, for instance, higher quality images may be possible(e.g., using an MRI if an MRI is not available during a treatmentsession, or a CT when only a CBCT is available prior to treatment), andmore time can be spent generating and validating a patient model. Instill another example, data from multiple patients may be used togenerate a more robust model, such as to avoid over constraining themodel.

FIG. 5 illustrates an exemplary flowchart showing a patient statedictionary generation technique. Generation of a dictionary for use witha machine learning algorithm to output a patient state estimation mayuse coupled potential patient states and measurements.

The technique for generating the dictionary includes an operation 502 toreceive a measurement or set of measurements, with a correspondingpatient state or corresponding set of patient states. For example, ameasurement may include a 2D image or other partial patient stateinformation or data. The patient state may include a 3D or 4Drepresentation of a patient, corresponding to the measurement. Thedictionary thus may include labeled data for training or testing themachine learning algorithm. In an example, the received measurements mayinclude digitally reconstructed radiograph (DRR) images for a CT-basedpatient model or 2D MRI slices for an MRI-based patient model.

In an example, generating the dictionary may include not using 2D imagesdirectly as measurements, but calculating a 2D DVF on the 2D images. Forexample, a PCA analysis of the 2D DVFs which results in a fewparameters. In this example, the input is the 2D PCA of 2D DVFparameters. In another example, real-time 2D images are registered to areference 2D image acquired during the same session (e.g., with samecontrast). This allows for the use of a fast, highly-parallelizable,deformable image registration technique to generate 2D DVFs, such as thedemon algorithm. The demon algorithm may be suitable for parallelimplementation in GPU with real-time performance. In yet anotherexample, a convolutional neural network (CNN) may be used to estimatethe 2D optical flow between two images in real-time to generate 2D DVFs.

The technique includes an operation 504 to generate a set of expandedpotential measurements and corresponding potential patient states. Thepotential measurements and potential patient states may be generated bytaking an initial actual measurement (e.g., one received in operation502) and a corresponding actual patient state and adding noise,perturbing, or otherwise extrapolating additional measurement-patientstate pairs that potentially may occur for a given patient or patients.The operation 504 allows for generation of a set of labeled data fromeven a single actual measurement and patient state pair.

The technique includes an operation 506 to save the set of expandedpotential measurements and corresponding potential patient states in adictionary for use with a machine learning technique. In an example, thereceived measurement or set of measurements and the receivedcorresponding patient state or set of patient states may also be savedin the dictionary for use with the machine learning technique. In anexample, a measurement (actual or potential) may be used as input datato the machine learning technique, with the corresponding patient state(actual or potential) being the output from the machine learningtechnique, and the correspondence acting as the label for the data.

The expanded potential measurements may be generated in operation 504using a low dimensional patient state representation (e.g., using PCA,ICA, CCA or the like). The low dimensional patient state representationmay be used to generate possible patient states that could potentiallyoccur during treatment. For example, a reasonable range of coefficients(e.g., a₁ and a₂ as described above in the DVF equation of thedescription of FIG. 3) may be subdivided into steps, such as equalsteps, and the resulting patient state may calculated for each step. Theresults may form the dictionary of potential states corresponding to thepotential measurement determined for each step. In an example, thecoefficients are sampled randomly such that they are representative ofmost likely motions that occur in actual patients. For example, aGaussian distribution, centered along the curve representing the averagerespiratory cycle, may be used to sample the coefficients. From theDVFs, the patient states (e.g. 3D patient images) are calculated bywarping a reference image. In an example, the dictionary may be used toinfer a 3D image from a 2D input, such as using regression analysis in asupervised machine learning algorithm.

In an example, to generate the set of expanded potential measurementsand corresponding potential patient states, small rigid transformationsmay be applied to a 3D image for data augmentation to for potentialpatient translations. In an example, the dictionary of potential patientstates may be generated from multiple patients rather than theparticular patient under treatment, using acquired 3D images,biomechanical models, or the like. In another example, using theparticular patient may be used for practical considerations to limit thedata needed and ensure that the data is relevant to the patient beingtreated.

Using the PCA approach to generating the expanded potential measurementsand potential patient states may include generating PCA coefficients. Togenerate realistic training patient states, coefficients may be randomlydrawn according to a normal distribution, for example centered along anaverage trajectory (e.g., within the 4D dataset of a received patientstate) with a standard distribution equivalent to a percentage of adynamic range of each coefficient (e.g., 10%). Next, a PCA-to-DVFreconstruction is performed. A full DVF may be reconstructed using therandomly generated PCA coefficients (e.g., 2-3 coefficients representingdegrees of freedom of the moving patient). The DVF is converted to a rawpatient state volume by warping the reference volume using the full DVF.A partial measurement is created from the raw patient state volume. Fora CT-based motion model, a 2D digitally reconstructed radiograph iscomputed from the raw patient state volume using, for example, aSiddon-Jacobs algorithm. For a MRI-based motion model, the 3D volume isresampled to extract a 2D MRI slice. Small rigid transformations areapplied to the 3D volume for data augmentation to account for smalldifferences in the patient position inter-fraction.

Together, the raw patient state volume output and the partialmeasurement are used together as a training sample to be saved to thedictionary (e.g., the input is the measurement and the output is thepatient state). This workflow may repeated for a number of trainingsamples, such as 1,000s of samples, or an optimized number of trainingsamples.

For each generated potential patient state, one or more potentialpatient measurements are simulated. These are measurements that maypotentially have resulted in the corresponding state. For example, for astate represented by a 3D MRI image, a 2D slice from a particularorientation and position (e.g., sagittal) that is expected to be usedduring treatment can be extracted. For a 3D CBCT image, a kV x-rayprojection can be simulated by ray-tracing through the 3D image andintegrating voxels along the ray lines, such as by using theSiddon-Jacobs algorithm, for a particular gantry angle. More complexalgorithms can be considered, such as using a Monte Carlo algorithm tosimulate what a realistic 2D kV x-ray image potentially produces,including effects such as scattering and beam hardening. Imagingproperties such as slice or gantry angle may be randomly sampled, orsampled uniformly, or fixed at known values. In some examples, aseparate AI algorithm (e.g., Generative Adversarial Networks (GAN)) maybe used to estimate measurements from patient states, particularly whenthe patient measurement cannot be easily calculated from the state(e.g., 2D MR slice measurements from 3D density patient stateinformation). In some examples, the dimensionality of the dictionary maybe further reduced by performing unsupervised dimensionality reduction(e.g., PCA, ICA, or CCA) on either the potential measurements or patientstates. In other examples, a demon algorithm (with image registration toa reference image) or a CNN may be used to generate DVFs for thepotential measurements.

FIG. 6 illustrates an exemplary regression model machine learning engine600 for use in estimating a patient state. Machine learning engine 600utilizes a training engine 602 and a estimation engine 604. Trainingengine 602 inputs historical transaction information 606 (e.g., patientmeasurements and corresponding patient states) into featuredetermination engine 608. The historical action information 606 may belabeled to indicate the correspondence between a measurement and apatient state.

Feature determination engine 608 determines one or more features 610from this historical information 606. Stated generally, features 610 area set of the information input and include information determined to bepredictive of a particular outcome. The features 610 may be determinedby hidden layers, in an example. The machine learning algorithm 612produces a correspondence motion model 620 based upon the features 610and the labels.

In the estimation engine 604, current action information 614 (e.g., acurrent patient measurement) may be input to the feature determinationengine 616. Feature determination engine 616 may determine features ofthe current information 614 to estimate a corresponding patient state.In some examples, feature determination engine 616 and 608 are the sameengine. Feature determination engine 616 produces feature vector 618,which is input into the model 620 to generate one or more criteriaweightings 622. The training engine 602 may operate in an offline mannerto train the model 620. The estimation engine 604, however, may bedesigned to operate in an online manner. It should be noted that themodel 620 may be periodically updated via additional training or userfeedback (e.g., additional, changed, or removed measurements or patientstates).

The machine learning algorithm 612 may be selected from among manydifferent potential supervised or unsupervised machine learningalgorithms. Examples of supervised learning algorithms includeartificial neural networks, Bayesian networks, instance-based learning,support vector machines, decision trees (e.g., Iterative Dichotomiser 3,C4.5, Classification and Regression Tree (CART), Chi-squared AutomaticInteraction Detector (CHAID), and the like), random forests, linearclassifiers, quadratic classifiers, k-nearest neighbor, linearregression, logistic regression, and hidden Markov models. Examples ofunsupervised learning algorithms include expectation-maximizationalgorithms, vector quantization, and information bottleneck method.Unsupervised models may not have a training engine 602.

In an example, a regression model is used and the model 620 is a vectorof coefficients corresponding to a learned importance for each of thefeatures in the vector of features 610, 618. The regression model isillustrated in block 624, showing an example linear regression. Themachine learning algorithm 612 is trained using a dictionary generatedas described herein. The machine learning algorithm 612 trains on howpatient measurements correspond to patient states. In an example, themachine learning algorithm 612 implements a regression problem (e.g.,linear, polynomial, regression trees, kernel density estimation, supportvector regression, random forests implementations, or the like). Theresulting training parameters define the patient state generator as acorrespondence motion model for the chosen machine learning algorithm.

In the conventional linac case, this training may be performedseparately for every possible gantry angle (e.g., with a one degreeincrement), since x-ray acquisition orientation may be constrained to anorthogonal angle with respect to the treatment beam. In the MR-linaccase, control may be given to a clinician on the 2D acquisition planefor position or orientation. Repeating cross-validation on training datawith different choice of 2D planes can reveal which 2D planes yield bestsurrogate information for a given patient/tumor site.

In some cases a patient measurement may be used to update the model 620.In some cases a calculation may be performed to determine whether thepatient measurement is consistent with the model 620, and pause thetreatment if it is not (e.g., using a threshold on the variance for aKDE algorithm, or determining whether there is sufficient data in thedictionary in the neighborhood of the measurement). When the treatmentis paused, a new model 620 may be generated, or the old model 620 may bereused if the measurement (e.g., motion) was an aberration.

In some applications the entire real-time patient image may not benecessary, and only features of it may be useful. For example the targetcentroid may be useful to make geometric corrections to multileafcollimators (MLCs), or to gate a beam on or off. In such cases, thesingle DVF vector connecting the center of the target in the referenceimage to the current target may be used rather than computing the entire3D DVF and deforming the entire reference image at each time, whichresults in rendering the real-time process more efficient.

After the patient state generator has been successfully trained and thepatient model 620 is aligned to the patient, the treatment beam isturned on and instantaneous partial measurements are acquired at a givenfrequency. For each received measurement, the process may includenormalizing a 2D image of the received measurement to match the contrastof training images. The patient state generator may use the normalizedmeasurement to infer model coefficients, and a DVF may be reconstructedusing the model. The reconstructed DVF is used to warp the referencevolume and treatment information to the current patient state, which maybe output or saved.

In some cases, the model may not be well-aligned to the patient duringtreatment. This may occur if the patient moves between the 4D image andtreatment, if a model from a previous day is used, or if data from otherpatients is used. The patient model (computed pre-treatment) may then bealigned to the actual patient position by rigid registration to newpatient measurements with the patient in treatment position. During thistime, a CBCT or MRI is acquired for coarse model-to-patient alignment.Fine alignment of patient model with multiple sample images (e.g., x-rayor 2D MRI slices) to account for couch shifts can be applied after CBCTor MRI acquisition.

Differences of contrast in synthetically generated training measurementsversus actual 2D imaging acquisitions can hinder the generator's abilityto infer 3D patient states. Some intensity normalization procedure maybe used to correct for this issue. For example, local or global linearnormalization methods may be used. Other examples may include the use ofa Generative Adversarial Network (GAN) for mapping the intensities ofreal versus synthetic images.

FIG. 7 illustrates a flowchart 700 of exemplary operations forestimating a patient state. The flowchart 700 includes an optionaloperation 702 to receive, for example using a processor, patient dataincluding a set of patient measurements and corresponding patientstates. The corresponding patient states may include a 3D or a 4Dpatient image, such as a 3D CT, a 3D CBCT, a 3D MRI, a 3D PET, a 3Dultrasound, a 4D CT, a 4D CBCT, a 4D MRI, a 4D PET, or a 4D ultrasoundimage. The patient measurements may include a 2D MRI slice, MRI k-spacedata, a 1D MRI navigator, a 2D MRI projection, x-ray 2D projection data,PET data, a 2D ultrasound slice, or the like. In some cases, detectorsmay be arranged to obtain patient measurements from multiple viewssimultaneously, for example with stereoscopic kV imaging, or frommultiple concurrent modalities, such as kV imaging combined with asurface camera. In one example, the patient data may be generated from asingle patient. In another example, the patient data may be include datafrom a plurality of patients.

The flowchart 700 includes an operation 704 to identify a preliminarymotion model of a patient under motion, for example based on the set ofpatient measurements and corresponding patient states. In an example,the preliminary motion model may be generated based on a 4D datasetacquired before a radiotherapy treatment. The preliminary motion modelmay be generated from a 4D MR or 4D CBCT acquired during treatment. ADVF may be calculated between each of the phases of this 4D image and areference phase. A PCA analysis may be performed on these DVFs. Thepreliminary motion model may be a 3D DVF that is parameterized by 2-3scalars, plus a reference 3D image. Potential 3D images may be generatedthat might occur during a treatment from the 2-3 scalars, by which a DVFmay be calculated. This DVF may be used to deform the reference 3D imageto calculate a new 3D image.

The flowchart 700 includes an operation 706 to generate a dictionary ofexpanded potential patient measurements and corresponding potentialpatient states using the motion model. The expanded potential patientmeasurements may include deformations of a 3D or a 4D patient image. Inan example, the deformations include deformation vector fields (DVFs)calculated using a deformable registration algorithm. In an example, theexpanded potential patient measurements include a 2D projection image.The expanded potential patient measurements may be generated using oneor more of extracting a 2D slice from a 3D image, ray-tracing through a3D image to generate a 2D projection image, simulating x-rayinteractions with a 3D image using a Monte Carlo technique, using acollapsed cone convolution technique, using a superposition andconvolution technique, using a generative adversarial network, using aconvolutional neural network, using a recurrent neural network or thelike. The dictionary may include possible 3D images that may occurduring treatment by randomly sampling 2-3 scalars, generating to 3D DVFsfrom the scalars, and deforming the reference image, resulting in thecorresponding potential patient states.

The expanded potential patient measurements may be generated bycalculating a 2D DVF on a 2D input image. In an example, the 2D DVF maybe calculated by performing a PCA analysis of the 2D input image. Inanother example, the 2D DVF may be calculated by registering the 2Dinput image to a reference 2D image (e.g., taken at the start or justbefore the start of radiation treatment), and using a deformable imageregistration technique. The 2D input image and the reference 2D imagemay have the same contrast to allow for registration. The deformableimage registration technique may be a fast, highly-parallelizabletechnique, such as a demon algorithm (e.g., implemented in parallel on aGPU with real-time performance). In yet another example, the 2D DVF maybe generated using a CNN to estimate a 2D optical flow between the 2Dinput image and a 2D reference image. The CNN may be run in real-time.

The corresponding potential patient states may be associated with apatient measurement that would have yielded respective patient states.For example, extracting a 2D slice through a 3D image, or a 2Dprojection through an image, at a particular location or angle. The raw2D image may not be used in an example, as the measurement. Instead, a2D DVF between the 2D image and a corresponding image form the reference3D image may be used with a PCA analysis on the resulting 2D DVF. Forexample, the measurements may be processed versions of measurements,rather than directly measured patient data. The measurements may be PCAcomponents of 2D DVFs, which may include the expanded potential patientmeasurements. The coupled expanded potential patient measurements (PCAsof 2D DVFs) and corresponding patient states (PCAs of 3D DVFs) may formthe dictionary.

The flowchart 700 includes an operation 708 to train, using a machinelearning technique, a correspondence motion model relating an inputpatient measurement to an output patient state using the dictionary. Thecorrespondence motion model may include a deformation vector field (DVF)as a function of one or more parameters. In an example, the one or moreparameters may be determined by reducing dimensionality of a preliminaryDVF calculated between two or more phases of a 4D image and a referencephase. For example, reducing the dimensionality may include using aprincipal component analysis (PCA), an independent component analysis(ICA) or a canonical correlation analysis (CCA). In an example, thecorrespondence motion model may be generated using a random forestregression, a linear regression, a polynomial regression, a regressiontree, a kernel density estimation, a support vector regressionalgorithm, a CNN, a RNN, or the like. The machine learning algorithm maybe used to relate the coupled entries in the dictionary. The algorithmmay be used with a measurement input to provide a patient state. Themeasurement input may include a PCA component of a 2D DVF of a 2D imagewith a reference image, and the patient state may include a 3D DVF.

In an example, the preceding operations occur pre-treatment, while thefollowing operations occur during treatment. The flowchart 700 includesan operation 710 to estimate the patient state corresponding to apatient measurement of the patient using the correspondence motionmodel. The patient state may be saved or output. For example, thepatient state may be output for display on a user interface of a displaydevice. In an example, estimating the patient state may includereceiving the patient measurement as an input to the correspondencemotion model, the input including a real-time stream of 2D images. Thereal-time stream of 2D images may include stereoscopic kV images (e.g.,from a kV imager rotating around a patient with a conventional linac) orpairs of 2D MR slice images (e.g., from an MR-Linac). In an example, thestereoscopic kV images may include two x-ray images that are orthogonalor substantially orthogonal (e.g., within 10 degrees) which are acquiredsimultaneously or substantially simultaneously (e.g., within a few or afew hundred milliseconds). The kV imager may be fixed in a room or maybe fixed to a gantry (e.g., including a linac). A pair of 2D MR sliceimages may be orthogonal to each other or parallel to each other. Inanother example, two kV imagers may be used, such as with each at 45degrees to the treatment beam (and 90 degrees to each other). Both kVimagers in this example may be used simultaneously or they may be usedin an alternating fashion.

In an example, images may be acquired from a kV imager or two kV imagerswith simultaneously acquired internal ultrasound data. The ultrasounddata may be used to reduce the kV dose by having, for example, less doseor pulse or to have a lower kV imaging frame rate. This secondary datamay be included directly into the measurements to calculate patientstate, or a separate correspondence model may be generated between thekV and secondary data streams and this separate model may be used torelate secondary data to patient state. For example, a correlation modelmay be built and continuously updated that relates kV PCA components toa parameter extracted from the secondary measurements stream, and as thesecondary stream data is acquired, this correlation model may be used todetermine the kV PCA components.

During treatment, a 2D image may be received. A 2D DVF may be calculatedbetween the input 2D image and a reference 2D image. A PCA analysis maybe performed on the DVF. The result is a real-time ‘measurement’ as usedherein. The trained machine learning algorithm may take the measurementas an input and calculate PCA components of the 3D DVF from the inputmeasurements. The PCA components are used to generate a 3D DVF, which isused to deform the 3D reference image with to 3D DVF to form the currentreal-time 3D patient image that represents the patient at a currenttime. The patient state may be the 3D image itself, the reference imageplus the 3D DVF, or the like (in an example, one may be calculated fromthe other).

In an example, an operation may include outputting the patient state,such as outputting two or more MR-like 3D images showing tissuecontrast, outputting non-imaging information, outputting CT-like 3Dimages, or the like.

FIG. 8 illustrates a flowchart of exemplary operations for performingradiation therapy techniques.

The flowchart 800 includes an operation 802 to generate a dictionary ofexpanded potential patient measurements and corresponding potentialpatient states using a motion model (e.g., as described above withrespect to operation 706). The expanded potential patient measurementsmay be generated by calculating a 2D DVF on a 2D input image. In anexample, the 2D DVF may be calculated by performing a PCA analysis ofthe 2D input image. In another example, the 2D DVF may be calculated byregistering the 2D input image to a reference 2D image (e.g., taken atthe start or just before the start of radiation treatment), and using adeformable image registration technique. The 2D input image and thereference 2D image may have the same contrast to allow for registration.The deformable image registration technique may be a fast,highly-parallelizable technique, such as a demon algorithm (e.g.,implemented in parallel on a GPU with real-time performance). In yetanother example, the 2D DVF may be generated using a CNN to estimate a2D optical flow between the 2D input image and a 2D reference image. TheCNN may be run in real-time.

In an example, the expanded potential patient measurements may begenerated from a 4D image, the 4D image including a 4D CT, a 4D CBCT, a4D MM, a 4D PET, a 4D ultrasound image, or the like. In an example, theexpanded potential patient measurements include a 2D projection imageand are generated by using at least one of: extracting a 2D slice from a3D image, ray-tracing through a 3D image to generate a 2D projectionimage, simulating x-ray interactions with a 3D image using a Monte Carlotechnique, using a collapsed cone convolution technique, using asuperposition and convolution technique, using a generative adversarialnetwork, a convolutional neural network, a recurrent neural network, orthe like.

The flowchart 800 includes an operation 804 to train, using a machinelearning technique, a correspondence motion model relating an inputpatient measurement to an output patient state using the dictionary(e.g., as described above with respect to operation 708). Thecorrespondence motion model may include a deformation vector field (DVF)as a function of one or more parameters, the one or more parametersdetermined by reducing dimensionality of a preliminary DVF calculatedbetween two or more phases of a 4D image and a reference phase. Thecorrespondence motion model may be generated using a random forestregression, a linear regression, a polynomial regression, a regressiontree, a kernel density estimation, a support vector regressionalgorithm, a convolutional neural network, a recurrent neural network,or the like.

The flowchart 800 includes an operation 806 to receive a real-timestream of 2D images from an image acquisition device (e.g., a kV x-ray,a MR device, a CT device, or other image acquisition device). Thereal-time stream of 2D images may include stereoscopic kV images (e.g.,from a kV imager rotating around a patient with a conventional linac) orpairs of 2D MR slice images (e.g., from an MR-Linac). In anotherexample, the real-time stream of 2D images may include k-space data, lowresolution 3D MR images, 1D navigators, or other MR information.

In an example, the stereoscopic kV images may include two x-ray imagesthat are orthogonal or substantially orthogonal (e.g., within 10degrees) which are acquired simultaneously or substantiallysimultaneously (e.g., within a few or a few hundred milliseconds). ThekV imager may be fixed in a room or may be fixed to a gantry (e.g.,including a linac). A pair of 2D MR slice images may be orthogonal toeach other or parallel to each other.

The flowchart 800 includes an operation 808 to estimate the patientstate corresponding to an image of the real-time stream of 2D imagesusing the correspondence motion model. The patient state may be output,for example as an image (e.g., a 3D MR or CT), as non-image information,or both. The patient state may include information (e.g., an image ortext) describing patient anatomy, such as a tumor or organ of interest,or may be used to establish a target, such as a radiation therapy target(e.g., on a portion of a tumor).

The flowchart 800 includes an operation 810 to locate a radiationtherapy target within a patient using the patient state.

The flowchart 800 includes an operation 812 to track a radiation therapytarget of a patient in real-time using the patient state. For example,successive images from the real-time stream of 2D images may be used tooutput corresponding patient states, with a target tracked from onepatient state to the next.

The flowchart 800 includes an operation 814 to direct radiation therapy,using a treatment device (e.g., a standalone treatment device, a devicecoupled to an image acquisition device (e.g., an MR-linac), or thelike), to a target according to the patient state. For example, thetarget may be located in operation 810 or tracked in operation 812, andradiation therapy may be applied according to the location or tracking.In an example, location or tracking information may be displayed on adisplay device, such as with a user interface presented on the displaydevice.

Additional Notes

The above detailed description includes references to the accompanyingdrawings, which form a part of the detailed description. The drawingsshow, by way of illustration but not by way of limitation, specificembodiments in which the invention can be practiced. These embodimentsare also referred to herein as “examples.” Such examples can includeelements in addition to those shown or described. However, the presentinventors also contemplate examples in which only those elements shownor described are provided. Moreover, the present inventors alsocontemplate examples using any combination or permutation of thoseelements shown or described (or one or more aspects thereof), eitherwith respect to a particular example (or one or more aspects thereof),or with respect to other examples (or one or more aspects thereof) shownor described herein.

All publications, patents, and patent documents referred to in thisdocument are incorporated by reference herein in their entirety, asthough individually incorporated by reference. In the event ofinconsistent usages between this document and those documents soincorporated by reference, the usage in the incorporated reference(s)should be considered supplementary to that of this document; forirreconcilable inconsistencies, the usage in this document controls.

In this document, the terms “a,” “an,” “the,” and “said” are used whenintroducing elements of aspects of the invention or in the embodimentsthereof, as is common in patent documents, to include one or more thanone or more of the elements, independent of any other instances orusages of “at least one” or “one or more.” In this document, the term“or” is used to refer to a nonexclusive or, such that “A or B” includes“A but not B,” “B but not A,” and “A and B,” unless otherwise indicated.

In the appended claims, the terms “including” and “in which” are used asthe plain-English equivalents of the respective terms “comprising” and“wherein.” Also, in the following claims, the terms “comprising,”“including,” and “having” are intended to be open-ended to mean thatthere may be additional elements other than the listed elements, suchthat after such a term (e.g., comprising, including, having) in a claimare still deemed to fall within the scope of that claim. Moreover, inthe following claims, the terms “first,” “second,” and “third,” etc.,are used merely as labels, and are not intended to impose numericalrequirements on their objects.

The present invention also relates to a computing system adapted,configured, or operated for performing the operations herein. Thissystem may be specially constructed for the required purposes, or it maycomprise a general purpose computer selectively activated orreconfigured by a computer program (e.g., instructions, code, etc.)stored in the computer. The order of execution or performance of theoperations in embodiments of the invention illustrated and describedherein is not essential, unless otherwise specified. That is, theoperations may be performed in any order, unless otherwise specified,and embodiments of the invention may include additional or feweroperations than those disclosed herein. For example, it is contemplatedthat executing or performing a particular operation before,contemporaneously with, or after another operation is within the scopeof aspects of the invention.

In view of the above, it will be seen that the several objects of theinvention are achieved and other advantageous results attained. Havingdescribed aspects of the invention in detail, it will be apparent thatmodifications and variations are possible without departing from thescope of aspects of the invention as defined in the appended claims. Asvarious changes could be made in the above constructions, products, andmethods without departing from the scope of aspects of the invention, itis intended that all matter contained in the above description and shownin the accompanying drawings shall be interpreted as illustrative andnot in a limiting sense.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described examples (or one or moreaspects thereof) may be used in combination with each other. Inaddition, many modifications may be made to adapt a particular situationor material to the teachings of the invention without departing from itsscope. While the dimensions, types of materials and example parameters,functions, and implementations described herein are intended to definethe parameters of the invention, they are by no means limiting and areexemplary embodiments. Many other embodiments will be apparent to thoseof skill in the art upon reviewing the above description. The scope ofthe invention should, therefore, be determined with reference to theappended claims, along with the full scope of equivalents to which suchclaims are entitled.

Also, in the above Detailed Description, various features may be groupedtogether to streamline the disclosure. This should not be interpreted asintending that an unclaimed disclosed feature is essential to any claim.Rather, inventive subject matter may lie in less than all features of aparticular disclosed embodiment. Thus, the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separate embodiment. The scope of the invention should bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

Each of these non-limiting examples may stand on its own, or may becombined in various permutations or combinations with one or more of theother examples.

Example 1 is a method for estimating a real-time patient state during aradiotherapy treatment, the method comprising: receiving, using aprocessor, patient data including a set of constructed patientmeasurements; identifying a preliminary motion model of a patient undermotion based on the set of constructed patient measurements; generatinga dictionary of expanded potential patient measurements andcorresponding potential patient states using the preliminary motionmodel; training, using a machine learning technique, a correspondencemotion model relating an input patient measurement to an output patientstate using the dictionary; and estimating, using the processor, thepatient state corresponding to a patient measurement of the patientusing the correspondence motion model.

In Example 2, the subject matter of Example 1 includes, wherein thecorresponding patient states include a 3D patient image.

In Example 3, the subject matter of Example 2 includes, wherein thecorresponding potential patient states include deformations of the 3Dpatient image.

In Example 4, the subject matter of Example 3 includes, wherein thedeformations include deformation vector fields (DVFs) calculated using adeformable registration algorithm.

In Example 5, the subject matter of Examples 1-4 includes, wherein thepatient measurements include a 2D MRI slice, MRI k-space data, a 1D MRInavigator, a 2D MRI projection, x-ray 2D projection data, PET data, or a2D ultrasound slice.

In Example 6, the subject matter of Examples 1-5 includes, wherein thepatient data includes a 4D image.

In Example 7, the subject matter of Example 6 includes, wherein the 4Dimage is a 4D CT, a 4D CBCT, a 4D MRI, a 4D PET, or a 4D ultrasoundimage.

In Example 8, the subject matter of Examples 1-7 includes, wherein thecorrespondence motion model includes a deformation vector field (DVF) asa function of one or more parameters, the one or more parametersdetermined by reducing dimensionality of a preliminary DVF calculatedbetween two or more phases of a 4D image and a reference phase.

In Example 9, the subject matter of Examples 1-8 includes, wherein theexpanded potential patient measurements include a 2D projection imageand are generated by using at least one of: extracting a 2D slice from a3D image, ray-tracing through a 3D image to generate a 2D projectionimage, simulating x-ray interactions with a 3D image using a Monte Carlotechnique, using a collapsed cone convolution technique, using asuperposition and convolution technique, using a generative adversarialnetwork, a convolutional neural network, or a recurrent neural network.

In Example 10, the subject matter of Examples 1-9 includes, wherein thecorrespondence motion model is generated using a random forestregression, a linear regression, a polynomial regression, a regressiontree, a kernel density estimation, a support vector regressionalgorithm, a convolutional neural network, or a recurrent neuralnetwork.

In Example 11, the subject matter of Examples 1-10 includes, whereinestimating the patient state corresponding to the patient measurementincludes receiving the patient measurement as an input to thecorrespondence motion model, the input including a real-time stream of2D images.

In Example 12, the subject matter of Example 11 includes, wherein thereal-time stream of 2D images includes stereoscopic kV images or pairsof 2D MR slice images.

In Example 13, the subject matter of Examples 1-12 includes, outputtingthe patient state as two or more MR-like 3D images showing tissuecontrast.

In Example 14, the subject matter of Examples 1-13 includes, wherein thepatient state includes non-imaging information.

In Example 15, the subject matter of Examples 1-14 includes, generatingthe preliminary motion model based on a 4D dataset acquired before theradiotherapy treatment.

In Example 16, the subject matter of Examples 1-15 includes, generatingthe constructed patient measurements by calculating a 2D deformationvector field (DVF) on a 2D input image.

In Example 17, the subject matter of Examples 1-16 includes, whereingenerating the constructed patient measurements includes performing aprincipal component analysis (PCA) analysis of the 2D 2D input image.

In Example 18, the subject matter of Examples 1-17 includes, whereingenerating the constructed patient measurements includes registering the2D input image to a reference 2D image, and using a deformable imageregistration technique to calculate the 2D DVF.

In Example 19, the subject matter of Examples 1-18 includes, whereingenerating the constructed patient measurements includes using aconvolutional neural network (CNN) to estimate a 2D optical flow betweenthe 2D input image and a 2D reference image to calculate the 2D DVF.

Example 20 is a system for estimating a patient state during aradiotherapy treatment, the system comprising: a processor coupled tomemory, the memory including instructions, which when executed by theprocessor, cause the processor to perform operations to: receive patientdata including a set of constructed patient measurements; identify apreliminary motion model of a patient under motion based on the set ofconstructed patient measurements; generate a dictionary of expandedpotential patient measurements and corresponding potential patientstates using the preliminary motion model; train, using a machinelearning technique, a correspondence motion model relating an inputpatient measurement to an output patient state using the dictionary; andestimate the patient state corresponding to a patient measurement of thepatient using the correspondence motion model.

Example 21 is a method for estimating a real-time patient state during aradiotherapy treatment using a magnetic resonance linear accelerator(MR-Linac), the method comprising: generating a dictionary of expandedpotential patient measurements and corresponding potential patientstates using a preliminary motion model; training, using a machinelearning technique, a correspondence motion model relating an inputpatient measurement to an output patient state using the dictionary;receiving a real-time stream of 2D MR images from an image acquisitiondevice; estimating, using the processor, the patient state correspondingto an image of the real-time stream of 2D MR images using thecorrespondence motion model; and directing radiation therapy, using atreatment device coupled to the image acquisition device, to a targetaccording to the patient state.

In Example 22, the subject matter of Example 21 includes, wherein theexpanded potential patient measurements include deformations of a 3Dpatient image, and wherein the deformations include deformation vectorfields (DVFs) calculated using a deformable registration algorithm.

In Example 23, the subject matter of Examples 21-22 includes, whereinthe expanded potential patient measurements are generated from a 4Dimage, the 4D image including a 4D CT, a 4D CBCT, a 4D MRI, a 4D PET, ora 4D ultrasound image.

In Example 24, the subject matter of Examples 21-23 includes, whereinthe correspondence motion model includes a deformation vector field(DVF) as a function of one or more parameters, the one or moreparameters determined by reducing dimensionality of a preliminary DVFcalculated between two or more phases of a 4D image and a referencephase.

In Example 25, the subject matter of Examples 21-24 includes, whereinthe expanded potential patient measurements include a 2D projectionimage and are generated by using at least one of: extracting a 2D slicefrom a 3D image, ray-tracing through a 3D image to generate a 2Dprojection image, simulating x-ray interactions with a 3D image using aMonte Carlo technique, using a collapsed cone convolution technique,using a superposition and convolution technique, using a generativeadversarial network, a convolutional neural network, or a recurrentneural network.

In Example 26, the subject matter of Examples 21-25 includes, whereinthe correspondence motion model is generated using a random forestregression, a linear regression, a polynomial regression, a regressiontree, a kernel density estimation, a support vector regressionalgorithm, a convolutional neural network, or a recurrent neuralnetwork.

In Example 27, the subject matter of Examples 21-26 includes, outputtingthe patient state as two or more MR-like 3D images showing tissuecontrast.

In Example 28, the subject matter of Examples 21-27 includes, generatingthe expanded potential patient measurements by calculating a 2Ddeformation vector field (DVF) on a 2D input image.

In Example 29, the subject matter of Examples 21-28 includes, whereingenerating the expanded potential patient measurements includesperforming a principal component analysis (PCA) analysis of the 2D inputimage.

In Example 30, the subject matter of Examples 21-29 includes, whereingenerating the expanded potential patient measurements includesregistering the 2D input image to a reference 2D image, and using adeformable image registration technique to calculate the 2D DVF.

In Example 31, the subject matter of Examples 21-30 includes, whereingenerating the expanded potential patient measurements includes using aconvolutional neural network (CNN) to estimate a 2D optical flow betweenthe 2D input image and a 2D reference image to calculate the 2D DVF.

Example 32 is a method for generating real-time target localizationdata, the method comprising: generating a dictionary of expandedpotential patient measurements and corresponding potential patientstates using a preliminary motion model; training, using a machinelearning technique, a correspondence motion model relating an inputpatient measurement to an output patient state using the dictionary;receiving a real-time stream of 2D images from an image acquisitiondevice; estimating, using the processor, the patient state correspondingto an image of the real-time stream of 2D images using thecorrespondence motion model; locating a radiation therapy target withina patient using the patient state; and outputting the location of theradiation therapy target on a display device.

In Example 33, the subject matter of Example 32 includes, wherein thereal-time stream of 2D images includes stereoscopic kV images or pairsof 2D MR slice images.

In Example 34, the subject matter of Examples 32-33 includes, whereinthe expanded potential patient measurements include deformations of a 3Dpatient image, and wherein the deformations include deformation vectorfields (DVFs) calculated using a deformable registration algorithm.

In Example 35, the subject matter of Examples 32-34 includes, whereinthe expanded potential patient measurements are generated from a 4Dimage, the 4D image including a 4D CT, a 4D CBCT, a 4D MRI, a 4D PET, ora 4D ultrasound image.

In Example 36, the subject matter of Examples 32-35 includes, whereinthe correspondence motion model includes a deformation vector field(DVF) as a function of one or more parameters, the one or moreparameters determined by reducing dimensionality of a preliminary DVFcalculated between two or more phases of a 4D image and a referencephase.

In Example 37, the subject matter of Examples 32-36 includes, whereinthe expanded potential patient measurements include a 2D projectionimage and are generated by using at least one of: extracting a 2D slicefrom a 3D image, ray-tracing through a 3D image to generate a 2Dprojection image, simulating x-ray interactions with a 3D image using aMonte Carlo technique, using a collapsed cone convolution technique,using a superposition and convolution technique, using a generativeadversarial network, a convolutional neural network, or a recurrentneural network.

In Example 38, the subject matter of Examples 32-37 includes, whereinthe correspondence motion model is generated using a random forestregression, a linear regression, a polynomial regression, a regressiontree, a kernel density estimation, a support vector regressionalgorithm, a convolutional neural network, or a recurrent neuralnetwork.

In Example 39, the subject matter of Examples 32-38 includes, outputtingthe patient state as two or more MR-like 3D images showing tissuecontrast.

In Example 40, the subject matter of Examples 32-39 includes, generatingthe expanded potential patient measurements by calculating a 2Ddeformation vector field (DVF) on a 2D input image.

In Example 41, the subject matter of Examples 32-40 includes, whereingenerating the expanded potential patient measurements includesperforming a principal component analysis (PCA) analysis of the 2D inputimage.

In Example 42, the subject matter of Examples 32-41 includes, whereingenerating the expanded potential patient measurements includesregistering the 2D input image to a reference 2D image, and using adeformable image registration technique to calculate the 2D DVF.

In Example 43, the subject matter of Examples 32-42 includes, whereingenerating the expanded potential patient measurements includes using aconvolutional neural network (CNN) to estimate a 2D optical flow betweenthe 2D input image and a 2D reference image to calculate the 2D DVF.

Example 44 is a method for real-time tracking of a target, the methodcomprising: generating a dictionary of expanded potential patientmeasurements and corresponding potential patient states using apreliminary motion model; training, using a machine learning technique,a correspondence motion model relating an input patient measurement toan output patient state using the dictionary; receiving a real-timestream of 2D images from an image acquisition device; estimating, usingthe processor, patient states corresponding to images in the real-timestream of 2D images using the correspondence motion model; tracking aradiation therapy target of a patient in real-time using the patientstates; and outputting tracking information for the radiation therapytarget for display on a display device.

In Example 45, the subject matter of Example 44 includes, wherein thereal-time stream of 2D images includes stereoscopic kV images or pairsof 2D MR slice images.

In Example 46, the subject matter of Examples 44-45 includes, whereinthe expanded potential patient measurements include deformations of a 3Dpatient image, and wherein the deformations include deformation vectorfields (DVFs) calculated using a deformable registration algorithm.

In Example 47, the subject matter of Examples 44-46 includes, whereinthe expanded potential patient measurements are generated from a 4Dimage, the 4D image including a 4D CT, a 4D CBCT, a 4D MRI, a 4D PET, ora 4D ultrasound image.

In Example 48, the subject matter of Examples 44-47 includes, whereinthe correspondence motion model includes a deformation vector field(DVF) as a function of one or more parameters, the one or moreparameters determined by reducing dimensionality of a preliminary DVFcalculated between two or more phases of a 4D image and a referencephase.

In Example 49, the subject matter of Examples 44-48 includes, whereinthe expanded potential patient measurements include a 2D projectionimage and are generated by using at least one of: extracting a 2D slicefrom a 3D image, ray-tracing through a 3D image to generate a 2Dprojection image, simulating x-ray interactions with a 3D image using aMonte Carlo technique, using a collapsed cone convolution technique,using a superposition and convolution technique, using a generativeadversarial network, a convolutional neural network, or a recurrentneural network.

In Example 50, the subject matter of Examples 44-49 includes, whereinthe correspondence motion model is generated using a random forestregression, a linear regression, a polynomial regression, a regressiontree, a kernel density estimation, a support vector regressionalgorithm, a convolutional neural network, or a recurrent neuralnetwork.

In Example 51, the subject matter of Examples 44-50 includes, outputtingthe patient state as two or more MR-like 3D images showing tissuecontrast.

In Example 52, the subject matter of Examples 44-51 includes, generatingthe expanded potential patient measurements by calculating a 2Ddeformation vector field (DVF) on a 2D input image.

In Example 53, the subject matter of Examples 44-52 includes, whereingenerating the expanded potential patient measurements includesperforming a principal component analysis (PCA) analysis of the 2D inputimage.

In Example 54, the subject matter of Examples 44-53 includes, whereingenerating the expanded potential patient measurements includesregistering the 2D input image to a reference 2D image, and using adeformable image registration technique to calculate the 2D DVF.

In Example 55, the subject matter of Examples 44-54 includes, whereingenerating the expanded potential patient measurements includes using aconvolutional neural network (CNN) to estimate a 2D optical flow betweenthe 2D input image and a 2D reference image to calculate the 2D DVF.

Example 56 is at least one machine-readable medium includinginstructions that, when executed by processing circuitry, cause theprocessing circuitry to perform operations to implement of any ofExamples 1-55.

Example 57 is an apparatus comprising means to implement of any ofExamples 1-55.

Example 58 is a system to implement of any of Examples 1-55.

Example 59 is a method to implement of any of Examples 1-55.

Method examples described herein may be machine or computer-implementedat least in part. Some examples may include a computer-readable mediumor machine-readable medium encoded with instructions operable toconfigure an electronic device to perform methods as described in theabove examples. An implementation of such methods may include code, suchas microcode, assembly language code, a higher-level language code, orthe like. Such code may include computer readable instructions forperforming various methods. The code may form portions of computerprogram products. Further, in an example, the code may be tangiblystored on one or more volatile, non-transitory, or non-volatile tangiblecomputer-readable media, such as during execution or at other times.Examples of these tangible computer-readable media may include, but arenot limited to, hard disks, removable magnetic disks, removable opticaldisks (e.g., compact disks and digital video disks), magnetic cassettes,memory cards or sticks, random access memories (RAMs), read onlymemories (ROMs), and the like.

What is claimed is:
 1. A method for estimating a real-time patient stateduring a radiotherapy treatment, the method comprising: identifying,using a processor, a preliminary motion model of a patient under motion;generating a dictionary of expanded potential patient measurements andcorresponding potential patient states using the preliminary motionmodel; training, using a machine learning technique, a correspondencemotion model relating an input patient measurement to an output patientstate using the dictionary; and estimating, using the processor, thepatient state corresponding to a patient measurement of the patientusing the correspondence motion model.
 2. The method of claim 1, whereinthe corresponding patient states include a 3D patient image.
 3. Themethod of claim 2, wherein the corresponding potential patient statesinclude deformations of the 3D patient image.
 4. The method of claim 3,wherein the deformations include deformation vector fields (DVFs)calculated using a deformable registration algorithm.
 5. The method ofclaim 4, wherein the DVFs are 3D DVFs applied to the correspondencemotion model to generate the patient state using deformation.
 6. Themethod of claim 3, wherein the deformations include parameterizations of3D DVFs and the preliminary motion model includes a reference image. 7.The method of claim 1, wherein the patient measurements include a 2D MRIslice, MRI k-space data, a 1D MRI navigator, a 2D MRI projection, x-ray2D projection data, PET data, or a 2D ultrasound slice.
 8. The method ofclaim 1, wherein the patient data includes a 4D image.
 9. The method ofclaim 8, wherein the 4D image is a 4D CT, a 4D CBCT, a 4D MRI, a 4D PET,or a 4D ultrasound image.
 10. The method of claim 1, wherein thecorrespondence motion model includes a deformation vector field (DVF) asa function of one or more parameters, the one or more parametersdetermined by reducing dimensionality of a preliminary DVF calculatedbetween two or more phases of a 4D image and a reference phase.
 11. Themethod of claim 1, wherein the expanded potential patient measurementsinclude a 2D projection image and are generated by using at least oneof: extracting a 2D slice from a 3D image, ray-tracing through a 3Dimage to generate a 2D projection image, simulating x-ray interactionswith a 3D image using a Monte Carlo technique, using a collapsed coneconvolution technique, using a superposition and convolution technique,using a generative adversarial network, a convolutional neural network,or a recurrent neural network.
 12. The method of claim 1, wherein thecorrespondence motion model is generated using a random forestregression, a linear regression, a polynomial regression, a regressiontree, a kernel density estimation, a support vector regressionalgorithm, a convolutional neural network, or a recurrent neuralnetwork.
 13. The method of claim 1, wherein estimating the patient statecorresponding to the patient measurement includes receiving the patientmeasurement as an input to the correspondence motion model, the inputincluding a real-time stream of 2D images.
 14. The method of claim 13,wherein the real-time stream of 2D images includes stereoscopic kVimages or pairs of 2D MR slice images.
 15. The method of claim 1,further comprising outputting the patient state as two or more MR-like3D images showing tissue contrast.
 16. The method of claim 1, whereinthe patient state includes non-imaging information.
 17. The method ofclaim 1, further comprising generating the preliminary motion modelbased on a 4D dataset acquired before the radiotherapy treatment. 18.The method of claim 1, further comprising generating the expandedpotential patient measurements by calculating a 2D deformation vectorfield (DVF) on a 2D input image.
 19. The method of claim 1, whereingenerating the expanded potential patient measurements includesperforming a principal component analysis (PCA) analysis of the 2D inputimage.
 20. The method of claim 1, wherein generating the expandedpotential patient measurements includes registering the 2D input imageto a reference 2D image, and using a deformable image registrationtechnique to calculate the 2D DVF.
 21. The method of claim 1, whereingenerating the expanded potential patient measurements includes using aconvolutional neural network (CNN) to estimate a 2D optical flow betweenthe 2D input image and a 2D reference image to calculate the 2D DVF. 22.A method for generating real-time target localization data, the methodcomprising: generating a dictionary of expanded potential patientmeasurements and corresponding potential patient states using apreliminary motion model; training, using a machine learning technique,a correspondence motion model relating an input patient measurement toan output patient state using the dictionary; receiving a real-timestream of 2D images from an image acquisition device; estimating, usingthe processor, the patient state corresponding to an image of thereal-time stream of 2D images using the correspondence motion model;locating a radiation therapy target within a patient using the patientstate; and outputting the location of the radiation therapy target on adisplay device.
 23. The method of claim 22, wherein the real-time streamof 2D images includes stereoscopic kV images or pairs of 2D MR sliceimages.
 24. The method of claim 22, wherein the expanded potentialpatient measurements include deformations of a 3D patient image, andwherein the deformations include deformation vector fields (DVFs)calculated using a deformable registration algorithm.
 25. The method ofclaim 22, wherein the expanded potential patient measurements aregenerated from a 4D image, the 4D image including a 4D CT, a 4D CBCT, a4D MRI, a 4D PET, or a 4D ultrasound image.
 26. The method of claim 22,wherein the correspondence motion model includes a deformation vectorfield (DVF) as a function of one or more parameters, the one or moreparameters determined by reducing dimensionality of a preliminary DVFcalculated between two or more phases of a 4D image and a referencephase.
 27. The method of claim 22, wherein the expanded potentialpatient measurements include a 2D projection image and are generated byusing at least one of: extracting a 2D slice from a 3D image,ray-tracing through a 3D image to generate a 2D projection image,simulating x-ray interactions with a 3D image using a Monte Carlotechnique, using a collapsed cone convolution technique, using asuperposition and convolution technique, using a generative adversarialnetwork, a convolutional neural network, or a recurrent neural network.28. The method of claim 22, wherein the correspondence motion model isgenerated using a random forest regression, a linear regression, apolynomial regression, a regression tree, a kernel density estimation, asupport vector regression algorithm, a convolutional neural network, ora recurrent neural network.
 29. The method of claim 22, furthercomprising outputting the patient state as two or more MR-like 3D imagesshowing tissue contrast.
 30. The method of claim 22, wherein generatingthe expanded potential patient measurements includes performing aprincipal component analysis (PCA) analysis of the 2D input image. 31.The method of claim 22, wherein generating the expanded potentialpatient measurements includes registering the 2D input image to areference 2D image, and using a deformable image registration techniqueto calculate the 2D DVF.
 32. The method of claim 22, wherein generatingthe expanded potential patient measurements includes using aconvolutional neural network (CNN) to estimate a 2D optical flow betweenthe 2D input image and a 2D reference image to calculate the 2D DVF. 33.A method for real-time tracking of a target, the method comprising:generating a dictionary of expanded potential patient measurements andcorresponding potential patient states using a preliminary motion model;training, using a machine learning technique, a correspondence motionmodel relating an input patient measurement to an output patient stateusing the dictionary; receiving a real-time stream of 2D images from animage acquisition device; estimating, using the processor, patientstates corresponding to images in the real-time stream of 2D imagesusing the correspondence motion model; tracking a radiation therapytarget of a patient in real-time using the patient states; andoutputting tracking information for the radiation therapy target fordisplay on a display device.
 34. The method of claim 33, wherein thereal-time stream of 2D images includes stereoscopic kV images or pairsof 2D MR slice images.
 35. The method of claim 33, further comprisinggenerating the expanded potential patient measurements by calculating a2D deformation vector field (DVF) on a 2D input image.
 36. The method ofclaim 33, wherein generating the expanded potential patient measurementsincludes performing a principal component analysis (PCA) analysis of the2D input image.
 37. The method of claim 33, wherein generating theexpanded potential patient measurements includes registering the 2Dinput image to a reference 2D image, and using a deformable imageregistration technique to calculate the 2D DVF.
 38. The method of claim33, wherein generating the expanded potential patient measurementsincludes using a convolutional neural network (CNN) to estimate a 2Doptical flow between the 2D input image and a 2D reference image tocalculate the 2D DVF.
 39. A method for generating real-time targetlocalization data, the method comprising: generating, using a processor,a 3D deformation vector field (DVF) parameterized by two or more scalarsfrom a 4D image by: calculating DVFs between each phase of the 4D imageand a reference image; and performing a principal component analysis(PCA) analysis on the DVFs; generating potential 3D images by: randomlysampling the two or more scalars; generating new 3D DVFs from therandomly sampled two or more scalars; and deforming the reference imageto generate corresponding patient states; associating the correspondingpatient states with respective potential patient measurementscorresponding to the randomly sampled two or more scalars; calculatingcorresponding 2D DVFs between the respective potential patientmeasurements and a portion of the reference image; performing a PCA onthe corresponding 2D DVFs resulting in expanded potential patientmeasurements; training, using a machine learning technique, acorrespondence motion model relating the expanded potential patientmeasurements to the corresponding patient states using a dictionary;receiving a 2D image of a patient during radiotherapy treatment; inreal-time, calculating a measurement by: calculating a 2D DVF betweenthe 2D image and a portion of the reference image; and performing a PCAon the 2D DVF; inputting the measurement to the correspondence motionmodel to generate PCA components of a reconstructed 3D DVF; generatingthe reconstructed 3D DVF from the PCA components; and deforming thereference image with the reconstructed 3D DVF to generate a currentreal-time 3D patient image.
 40. The method of claim 39, furthercomprising outputting a current real-time patient state comprising thecurrent real-time 3D patient state or the reference image and thereconstructed 3D DVF.
 41. The method of claim 39, wherein the respectivepotential patient measurements include a 2D image extracted from a 3Dimage or a 2D projection through a 3D image.
 42. The method of claim 39,wherein the 4D image is a 4D MR image or a 4D CBCT image.